Automated object recognition kiosk for retail checkouts

ABSTRACT

A system, method, and apparatus for automated object recognition and checkout at a retail kiosk is provided. The system includes a controller configured with a processor and a memory to control operations of the automated retail checkout system. The system further includes an imaging device in communication with the controller and configured to create multiple electronic images of an object, such as a product for purchase. The system also includes an object recognition device in communication with the controller and the imaging device. The processor may execute software to receive electronic images from the imaging device, extract at least one feature from the images, and recognize the object based on a predetermined model being applied to the extracted feature from the images. The system also includes a display device to display an indication of the recognized object from the object recognition device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation in part of U.S. application Ser. No.16/168,066 filed 23 Oct. 2018, which is a continuation of U.S.application Ser. No. 14/517,634, filed 17 Oct. 2014, which claims thebenefit of U.S. Provisional Application No. 61/891,902 filed 17 Oct.2013, each of which are incorporated in their entireties by thisreference.

TECHNICAL FIELD

The presently disclosed embodiments relate to retail stores, and moreparticularly to an object recognition kiosk for retail checkouts.

BACKGROUND

Retail outlets (e.g., supermarkets, cafeterias, etc.) offer sale ofvarious products and services. The outlets are typically equipped withself-checkout kiosks that allow a shopper to scan the products selectedfor purchase on their own to receive indicia of their prices. Theshopper may then use the indicia to make a payment for completing theproduct purchase.

The products are available either as packaged items or fresh items forpurchase. The packaged items typically carry identification markers suchas bar codes and radio frequency identification (RFID) tags, which arescanned by relevant scanners equipped with the self-checkout kiosks.However, the fresh items (e.g., freshly cooked meals such as differenttypes of curries, pastas, and breads; various salads; fresh fruits andvegetables; etc.) are often untagged and/or unpacked, and require astore attendant to intervene for enabling their purchase. The storeattendant traditionally uses his personal assessment of the type andnumber of ingredients for each fresh item and manually inputs theassessed information to a checkout kiosk for expected payment tocomplete the purchase.

Since the collection of fresh items at the retail outlets may changebased on customer demand or product offerings, the assessed informationmay vary based on related inventory knowledge and skill of the storeattendant. As a result, the assessed information may become susceptibleto error and hence business loss. The probability of erroneousassessment increases when various fresh items are mixed together basedon customer request or as a new product offering. Such assistedcheckouts for fresh items may also become labor intensive and timeconsuming based on the quantity of fresh items being checked out.Further, customer queues may become bottlenecks during peak periods ofcustomer demand, possibly to provoke the customers to leave the retailoutlet to shop elsewhere. Other sales may be lost from customers who maysimply avoid a retail location at known busy times and shop elsewhere,because of past inconvenience from delays.

Therefore, there exists a need for an automated object recognition kioskfor retail checkout of fresh foods and provide a seamless retailcheckout experience for better customer service.

SUMMARY

In view of the deficiencies in the conventional methodologies for retailcheckout at a kiosk, the disclosed subject matter provides a system,method, and apparatus for automated object recognition and checkout at aretail kiosk.

According to one aspect of the disclosed subject matter, a system forautomated retail checkout is provided. In an aspect, a controller of thesystem can be configured with a processing system and a memory tocontrol operations of the automated retail checkout system. In otheraspects, an imaging device can be in communication with the controllerand configured to create one or more electronic images of an object,such as a product for purchase. In further aspects, an objectrecognition device can be in communication with the controller and theimaging device. The object recognition device can be configured with aprocessing system executing software to receive electronic images fromthe imaging device, extract at least one feature from the one or more ofthe images, and recognize the object based on a predetermined model ofobjects from an object database being applied to the feature from theone or more images. In another aspect, a display device can beconfigured with the system to display an indication from the objectrecognition device of the recognized object.

According to one embodiment of the disclosed subject matter, the atleast one feature extracted from the one or more images by the softwarecan be used by the processing system to train the object recognitiondevice using a predetermined machine learning method that formulates themodel based on recognizing the at least one feature from the object. Inanother embodiment, an illumination device can be configured by thecontroller to generate light having a predetermined level of brightnessand to illuminate the object using the generated light. In yet anotherembodiment, the object recognition device measures a change in lightingfrom a calibration pattern as perceived by the imaging device after theelectronic image of the object is created by the imaging device.According to another embodiment, the imaging device comprises a group ofmaneuverable cameras, and the controller can automatically calibratepositions of the group of cameras relative to the object based on thecalibration pattern. According to still another embodiment, the objectrecognition device analyzes the one or more electronic images from theimaging device and tracks a movement of a support structure incommunication with the object. In further embodiments, the controllercan adaptively tune the illumination device to generate light toilluminate the object based on the calibrated positions of the camerasand the position of the object. According to another embodiment, aweight sensor can be in communication with the controller and configuredto measure weight of the object.

According to another aspect of the disclosed subject matter,computer-implemented methodology for purchasing a product with a retailcheckout apparatus is provided. In an aspect, a methodology forcontrolling operations of the retail checkout apparatus with a computerincludes providing a processing system for executing softwareinstructions for illuminating, with an illumination device having apredetermined level of brightness controlled by the computer, apredetermined region of the retail checkout apparatus. The methodologyfurther includes capturing, with an imaging device controlled by thecomputer, one or more images of a product located within thepredetermined region; and recognizing, by the computer, an identity ofthe product based on a predetermined model being applied to the capturedone or more images. In another aspect, the methodology includesproviding, by the computer, an indication of the recognized productbased on one or more predefined attributes of the determined product. Inone aspect, the methodology includes displaying, by the computer on adisplay interface, at least a portion of the provided indication forcompleting a purchase of the product.

According to another aspect of the disclosed subject matter, anapparatus for retail checkouts is provided. In an aspect, a head portionof the apparatus includes an illumination device and an imaging device.In another aspect, a base portion can be oriented a predetermineddistance below the head portion to create an object examination spacebetween the head portion and base portion. In other aspects, theillumination device can be configured to generate light within theobject examination space having a predetermined level of brightness andilluminate the object using the generated light, and the imaging devicecan be configured to create one or more electronic images of theilluminated object within the object examination space. In anotheraspect of the disclosure, an electronics portion, operationallyconnected to the imaging device and the illumination device, can includea processing system programmed with software to execute instructions toreceive the one or more electronic images from the imaging device. Theprocessing system is further programmed with the software to extract atleast one feature from the one or more images of the object, andrecognize the object based on a predetermined model of objects from anobject database being applied to the feature from the one or moreelectronic images. In another aspect of the apparatus, a display devicecan be operationally connected to the head portion and configured todisplay an indication of the object recognition from the software.

Other and further aspects and features of the disclosure will be evidentfrom reading the following detailed description of the embodiments,which are intended to illustrate, not limit, the present disclosure.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic that illustrates an exemplary object recognitionsystem, according to an embodiment of the present disclosure.

FIG. 2 is a perspective view of an exemplary automated objectrecognition kiosk, according to an embodiment of the present disclosure.

FIG. 3 is a front view of the exemplary automated object recognitionkiosk of FIG. 2, according to an embodiment of the present disclosure.

FIG. 4 is a portion of the exemplary automated object recognition kioskof FIG. 2, according to an embodiment of the present disclosure.

FIG. 5 depicts an example of the system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following detailed description is made with reference to thefigures. Preferred embodiments are described to illustrate thedisclosure, not to limit its scope, which is defined by the claims.Those of ordinary skill in the art will recognize a number of equivalentvariations in the description that follows.

1. Definitions

A “feature” is used in the present disclosure in the context of itsbroadest definition. The feature may refer to a property of an entitysuch as an image or an object. Examples of the property may include, butnot limited to, size, shape, brightness, color, and texture.

A “model” or “equation” is used in the present disclosure in the contextof its broadest definition. The model may refer to a mathematicalrepresentation involving one or more parameters, each of which maycorrespond to the feature.

2. Exemplary Embodiments

FIG. 1 is a schematic that illustrates an exemplary object recognitionsystem 100, according to an embodiment of the present disclosure. Someembodiments are disclosed in the context of an automated objectrecognition kiosk for retail checkout, e.g., in a cafeteria involvingrecognition of fresh foods including, but not limited to, fresh fruitsand vegetables, dairy products, freshly prepared eatables such ascurries, breads, pastas, salads, and burgers; or any combinationthereof. However, other embodiments may be applied in the context ofvarious business scenarios involving object recognition. Examples ofsuch scenarios may include, but not limited to, self-checkout ofproducts by customers in a supermarket, fast food restaurants, or coffeeshops; multi-product packaging of diversified products in a packagingplant; product quality control in a manufacturing plant; advanced driverassistance systems such as automatic parking systems; publicsurveillance systems; and automatic teller machines (ATMs).

The object recognition system 100 may represent any of a wide variety ofdevices capable of providing automated object recognition services tovarious devices. The object recognition system 100 may be implemented asa standalone and dedicated “black box” including hardware and installedsoftware, where the hardware is closely matched to the requirementsand/or functionality of the software. In some embodiments, the objectrecognition system 100 may enhance or increase the functionality and/orcapacity of a network to which it may be connected. The objectrecognition system 100 of some embodiments may include software,firmware, or other resources that support remote administration,operation, and/or maintenance of the object recognition system 100.

In one embodiment, the object recognition system 100 may be implementedas or in communication with any of a variety of computing devices (e.g.,a desktop PC, a personal digital assistant (PDA), a server, a mainframecomputer, a mobile computing device (e.g., mobile phones, laptops,etc.), an internet appliance, etc.). In some embodiments, the objectrecognition system 100 may be integrated with or implemented as awearable device including, but not limited to, a fashion accessory(e.g., a wrist band, a ring, etc.), a utility device (a hand-held baton,a pen, an umbrella, a watch, etc.), a body clothing, or any combinationthereof.

Other embodiments may include the object recognition system 100 beingimplemented by way of a single device (e.g., a computing device,processor or an electronic storage device 106) or a combination ofmultiple devices. The object recognition system 100 may be implementedin hardware or a suitable combination of hardware and software. The“hardware” may comprise a combination of discrete components, anintegrated circuit, an application-specific integrated circuit, a fieldprogrammable gate array, a digital signal processor, or other suitablehardware. The “software” may comprise one or more objects, agents,threads, lines of code, subroutines, separate software applications, twoor more lines of code or other suitable software structures operating inone or more software applications.

As illustrated, the object recognition system 100 may include acontroller 102 in communication with, or integrated with, interface(s)104, a storage device 106, an object recognition device 108, and/or aprocessing system 530. The controller 102 may execute machine readableprogram instructions for processing data (e.g., video data, audio data,textual data, etc.) and instructions received from one or more devicessuch as the object recognition device 108, and so on. The controller 102may include, for example, microprocessor, microcomputers,microcontrollers, digital signal processors, central processing units,state machines, logic circuits, and/or any devices that may manipulatesignals based on operational instructions. Among other capabilities, thecontroller 102 may be configured to fetch and execute computer readableinstructions in the storage device 106 associated with the objectrecognition system 100. In some embodiments, the controller 102 may beconfigured to convert communications, which may include instructions,queries, data, etc., from one or more devices such as the objectrecognition device 108 into appropriate formats to make thesecommunications compatible with a third-party data application, networkdevices, or interfaces such as output devices, and vice versa.Consequently, the controller 102 may allow implementation of the storagedevice 106 using different technologies or by different organizations,e.g., a third-party vendor, managing the storage device 106 using aproprietary technology. In some other embodiments, the controller 102may comprise or implement one or more real time protocols (e.g., sessioninitiation protocol (SIP), H.261, H.263, H.264, H.323, etc.) andnon-real time protocols known in the art, related art, or developedlater to facilitate communication with one or more devices.

The processing system 530 can function to process the set of images todetermine the item class. All or a portion of the processing system ispreferably local to the kiosk, but can alternatively be remote (e.g., aremote computing system), distributed between the local and remotesystem, distributed between multiple local systems, distributed betweenmultiple kiosks, and/or otherwise configured. The processing systempreferably includes one or more processors (e.g., CPU, GPU, TPU,microprocessors, etc.). The processing system can optionally includememory (e.g., RAM, flash memory, etc.) or other nonvolatile computermedium configured to store instructions for method execution,repositories, and/or other data. When the processing system is remote ordistributed, the system can optionally include one or more communicationmodules, such as long-range communication modules (e.g., cellular,internet, Wi-Fi, etc.), short range communication modules (e.g.,Bluetooth, Zigbee, etc.), local area network modules (e.g., coaxialcable, Ethernet, WiFi, etc.), and/or other communication modules.

The object recognition system 100 may include a variety of known,related art, or later developed interface(s) 104, including softwareinterfaces (e.g., an application programming interface, a graphical userinterface, etc.); hardware interfaces (e.g., cable connectors, akeyboard, a card reader, a barcode reader, a biometric scanner, aninteractive display screen, a printer, etc.); or both. The interface(s)104 may facilitate communication between various devices such as thecontroller 102, the storage device 106, and the object recognitiondevice 108 within the object recognition system 100.

In some embodiments, the interface(s) 104 may facilitate communicationwith other devices capable of interacting with the object recognitionsystem 100 over a network (not shown). The network may include, forexample, one or more of the Internet, Wide Area Networks (WANs), LocalArea Networks (LANs), analog or digital wired and wireless telephonenetworks (e.g., a PSTN, Integrated Services Digital Network (ISDN), acellular network, and Digital Subscriber Line (xDSL)), radio,television, cable, satellite, and/or any other delivery or tunnelingmechanism for carrying data. Network may include multiple networks orsub-networks, each of which may include, for example, a wired orwireless data pathway. The network may include a circuit-switched voicenetwork, a packet-switched data network, or any other network able tocarry electronic communications. For example, the network may includenetworks based on the Internet protocol (IP) or asynchronous transfermode (ATM), and may support voice using, for example, VoiP,Voice-over-ATM, or other comparable protocols used for voice, video, anddata communications.

The storage device 106 may be configured to store, manage, or process atleast one of (1) data in a database related to the object being detectedor recognized, and (2) a log of profiles of various devices coupled tothe controller 102 and associated communications including instructions,queries, data, and related metadata. The storage device 106 may compriseof any computer-readable medium known in the art, related art, ordeveloped later including, for example, volatile memory (e.g., RAM),non-volatile memory (e.g., flash, etc.), disk drive, etc., or anycombination thereof. Examples of the storage device 106 may include, butnot limited to, a storage server, a portable storage device (e.g., a USBdrive, an external hard drive, etc.), and so on. The server may beimplemented as any of a variety of computing devices including, forexample, a general purpose computing device, multiple networked servers(arranged in clusters or as a server farm), a mainframe, or so forth.

The object recognition device 108 may be configured to recognize anobject using various computer vision and machine learning methods knownin the art, related art, or developed later based on various attributesincluding, but not limited to, shape, size, texture, and color of theobject. The object recognition device 108 may include and/or communicatewith an illumination device 110 and may include and/or communicate withan imaging device 112. The illumination device no (e.g., compactfluorescent tubes, bulbs, light emitting diodes, etc.) may be configuredto substantially illuminate the object for being recognized by theobject recognition device 108. The imaging device 112 (e.g., a camera, alaser scanner, etc.) may be configured to create or capture an image ofthe illuminated object to be recognized. The created or captured imagemay be processed by the object recognition device 108 to recognize theobject being scanned by the imaging device 112.

The object recognition device 108 may receive multiple such objectimages from the database in the storage device 106, or the imagingdevice 112, or both, as a training dataset corresponding to a variety ofobjects for training the object recognition system 100 so that one ormore images of an object scanned or captured by the imaging device 112are analyzed and recognized by the object recognition device 108.Various features may be extracted from the training dataset. Examples ofthe features may include, but not limited to, shape, size, color,texture, and so on related to the object. The object recognition device108 may apply various known in the art, related art, or developed latermachine learning methods including supervised learning methods (e.g.,Gaussian process regression, Naive Bayes classifier, conditional randomfield, etc.); unsupervised learning methods (e.g.,expectation-maximization algorithm, vector quantization, generativetopographic map, information bottleneck method, etc.); andsemi-supervised learning methods (e.g., generative models, low-densityseparation, graph-based methods, heuristic approaches, etc.) to thetraining dataset for formulating one or more optimized models forrecognizing the objects. During operation, the object recognition device108 may apply the optimized models to the object images received fromthe imaging device 112 to recognize the corresponding objects.

FIG. 2 is a perspective view of an exemplary automated objectrecognition kiosk, according to an embodiment of the present disclosure.In one embodiment, the automated object recognition kiosk 200 mayimplement the object recognition system 100 for retail checkouts. Thekiosk 200 may include a head portion 202, a base portion 204, and asupport panel portion 206 configured to support the head portion 202 andthe base portion 204 of the kiosk 200. The head portion 202, the baseportion 204, and the support panel portion 206 may be made of any rigidand durable material known in the art, related art, or developed laterincluding metals, alloys, composites, and so on, or any combinationthereof, capable of withstanding heat generated by the electronicsintegrated with the kiosk components.

In one embodiment, the head portion 202 may include a top surfaceportion 208, a bottom surface portion 210, and a compartment betweenthem. The compartment may be configured to receive or embed hardwareelectronics. The top surface portion 208 may include a flat portion andan inclined portion having a predetermined slope relative to the flatportion. In one example, the slope may be substantially perpendicular tothe descending line of sight of a user on the inclined portion. Thecompartment may secure the illumination device 110 and the imagingdevice 112. The bottom surface portion 210 may be located opposite tothe base 204 may be substantially flat to avoid shadows being createddue to relative variation in the bottom surface portion 210. Further,the bottom surface portion 210 of the head portion 202 may be locatedopposite to the base portion 204 of the object recognition kiosk 200.The bottom surface portion 210 may be capable of passing light generatedby the illumination device 110 on to the base portion 204 of the objectrecognition kiosk 200. The bottom surface portion 210 may be made up ofor coated with any of the anti-glare materials known in the art, relatedart, or developed later to evenly project light on the object to berecognized. Such coated bottom surface portion 210 may minimize theprojection of shadows due to reflection of illuminated light from theobject and its surroundings. The shadows need to be minimized so thatthe object data (e.g., an object image and its attributes such as color,brightness, texture, etc.) as gathered by the imaging device 112 may beoptimally separated from a predetermined background such as the baseportion 204 by the implemented computer vision and machine learningmethods.

The head portion 202 may include side surfaces such as a side surface212 in communication with lateral edges of the top surface portion 208and the bottom surface portion 210 of the head 202. The side surfacesmay facilitate to reduce unwanted dissipation of generated light intothe ambient surrounding and to focus the generated light on to the baseportion 204 of the object recognition kiosk 200.

In one embodiment, the head portion 202 may be divided into a first part214 and a second part 216, each having a respective top surface 208, abottom surface 210 and a compartment for housing the correspondingelectronic components such as the illumination device no and the imagingdevice 112. The first part 214 and the second part 216 may have apredetermined spacing 218 between them to support electronics forseparate operation based on predetermined aesthetics of the head portion202. At least one of the first part 214 and the second part 216 mayinclude a display device such as an interactive display screen 220 tointeract with a user. Dimensions of the first part 214 may be similar tothe dimensions of the second part 216. However, the relative dimensionsof the first part 214 and the second part 216 may differ from each otherin some embodiments. In further embodiments, the head portion 202 may beintegrated with a variety of payment devices known in the art, relatedart, or developed later. For example, the second part 216 may include apredetermined card reader 220 to receive payments based on the objectbeing recognized by the object recognition kiosk 200. Both the firstpart 214 and the second part 216 may be secured to the support panelportion 206 using various known in the art, related art, or developedlater fastening techniques including a nut and screw arrangement,welding, push-on joint sockets, and so on.

The base portion 204 may refer to any surface, which may be sufficientlyilluminated by the light projected from the head portion 202 of theobject recognition kiosk 200. In some embodiments, the base portion 204may be coated with the anti-glare material for minimizing shadowprojections on the object. In the illustrated embodiment, the baseportion 204 may be coupled to the support panel portion 206 below thehead portion 202 of the object recognition kiosk 200. The base portion204 may have a substantially flat surface opposite to the bottom surfaceportion 210 of the head portion 202 so that an image of the objectplaced on the base portion 204 may be appropriately captured by theimaging device 112. In some embodiments, the base portion 204 may be anelevated surface from the ground and substantially parallel to thebottom surface portion 210. In some other embodiments, a predeterminedregion may be marked or relatively indented uniformly on the baseportion 204 to indicate that the predetermined region is capable ofbeing sufficiently illuminated by the illumination device noirrespective of ambient lighting conditions. The base portion 204 may besubstantially separated by a predefined distance from the head portion202 for accommodating at least one object in a space, hereinafterreferred to as an examination space 224, between the base portion 204and the head portion 202.

The front side 226 of the examination space 224 may be kept open toallow placement of objects. Rest of the sides of the examination space224 may be left partially or fully open depending on the ambientlighting conditions in which the kiosk 200 is used so that most of thelighting may be provided internally through the kiosk's own lightingsystem such as the illumination device 110. Some tolerance for externalambient lighting may be achieved using a calibration pattern 520 for thebase portion 204 and/or by adjusting various camera properties such asexposure, white balance, and gain.

The calibration pattern may include various colors such as red, green,blue, white, black and their shades or combinations. The calibrationpattern may be implemented as a software program in a computer readablemedium such as a smartcard, which may be integrated, or incommunication, with the object recognition kiosk 200 and used by theobject recognition device 108 to measure the change in ambient lightingand the effect of this lighting change on colors perceived by theimaging device 112. The calibration pattern may also be used by thecontroller 102 to determine the exact position of the imaging devices(e.g., the imaging device 112) relative to the base portion 204, to theobject, and/or to each other. The calibration pattern may be in anyshape such as squares, color wheel or just smeared in any kind of shapeinto the base portion 204.

FIG. 3 is a front view of the exemplary automated object recognitionkiosk 200 of FIG. 2, according to an embodiment of the presentdisclosure. In one embodiment, the support panel portion 206 may includeone or more openings for securing at least one imaging device 112 tocapture an image of the object held between the base portion 204 and thehead portion 202. In the illustrated example, the support panel portion206 may include a first opening 302 securing a first imaging device 304and a second opening 306 securing a second imaging device 308. In someembodiments, at least one of the first imaging device 304 and the secondimaging device 308 may behave as a tracking imaging device to track themovement of a support structure such as a human hand temporarily incommunication with the object for introducing the object to berecognized within the examination space 224 between the base portion 204and the bottom surface portion 210 of the head portion 202. The objectrecognition device 108 analyzes images created or captured by thetracking imaging device to track the movement of the support structure.Other embodiments may include the base portion 204 having one or moremeasurement sensors such as a weight sensor 310 for determining theweight of an object to be recognized upon being placed on the baseportion 204.

Further, the support panel portion 206 may have a slit 312 perpendicularto the spacing 218 between the first part 214 and the second part 216 ofthe head portion 202. The slit 312 may extend along the longitudinalaxis of the support panel portion 206 from a first end of the supportpanel portion 206 to the mid of the support panel portion 206. The firstend of the support panel portion 206 may be adjacent to the head portion202 of the object recognition kiosk 200. The slit 312 may facilitateincorporation of electronics separately for the first imaging device 304and the second imaging device 308 and may support aesthetics of theobject recognition kiosk 200.

FIG. 4 is a portion of the exemplary automated object recognition kiosk200 of FIG. 2, according to an embodiment of the present disclosure. Theillustrated embodiment shows respective compartments in each of thefirst part 214 and the second part 216 of the head portion 202 uponbeing viewed from the bottom surface portion 210. Each of thecompartments may include an imaging region and an illumination region.In one embodiment, the imaging region may be a relatively narrow regiondefined substantially along the edges of the first part 214 and thesecond part 216. The illumination region may be a region surrounded bythe imaging region. The illumination region may have a dimensionsubstantially greater than the dimension of the imaging region.

The imaging region may be configured to secure one or more imagingdevices and the illumination region configured to secure one or moreillumination devices. For example, an imaging region 402 of the firstpart 214 may include imaging devices such as cameras 404-1, 404-2, . . ., 404-n (collectively, cameras 404) and an imaging region 406 of thesecond part 216 may include imaging devices such as cameras 408-1,408-2, . . . , 408-n (collectively, cameras 408). Similarly, a firstillumination region 410 corresponding to the first part 214 may includethe illumination devices such as light emitting diode (LED) lights412-1, 412-2, . . . , 412-n (collectively, LED lights 412) and a secondillumination region 414 corresponding to the second part 216 may includethe illumination devices such as LED lights 416-1, 416-2, . . . , 416-n(collectively, LED lights 416). In a first example, the cameras 404, 408may be two-dimensional cameras (2D cameras) or three-dimensional cameras(3D cameras), or any combination thereof. The 2D cameras may be used tocollect image sequences of objects from multiple viewpoints, and 3Dcameras may be used to get 3D point cloud of objects. Multipleviewpoints facilitate to overcome occlusion as the far side of an objectmay not be visible to an individual camera, or in case there aremultiple objects on the base portion 204 of the kiosk 200, with somepartially or fully hidden from the view of an individual camera. Thecamera properties such as exposure, white balance, gain, focus, pan,tilt, saturation and others may be carefully determined and usuallypre-set during the operation life of the kiosk 200. These cameraproperties may be predefined to values such that changes to the ambientlighting conditions may be partially compensated by adjusting the valuesof these properties.

In a second example, the cameras 404, 408 may be a color video camerasuch as an HD webcam with at least one imaging channel for capturingcolor values for pixels corresponding generally to the primary visiblecolors (typically RGB). In a third example, the cameras 404, 408 may beinfrared cameras with at least one imaging channel for measuring pixelintensity values in the near-infrared (NIR) wavelength range. In afourth example, the cameras 404, 408 may be hybrid devices capable ofcapturing both color and NIR video. In a fifth example, the cameras 404,408 may be multi/hyperspectral cameras capable of capturing images atmultiple wavelength bands.

The cameras 404, 408 may be configured with at least one of the adaptivesteering technology and the controlled steering technology known in theart, related art, or developed later for maneuvering the direction ofthe imaging device 112 for capturing images based on the position of theobject within the examination space 224. Further, the intensity of theLED lights 412, 416 may be sufficiently high so that the ambient lightreceived by the examination space 224 and/or the base portion 204 isminimal. The light generated by the LED lights 412, 416 may besubstantially white light so that colors of the objects to be recognizedmay be optimally visible and captured by the cameras 404, 408.

FIG. 5 depicts an example of the automated object recognition kiosk 200.The automated object recognition kiosk 200 functions to sample images ofthe items. The kiosk can include: a kiosk housing defining anexamination space 224, and a set of sensors 580 monitoring theexamination space 224 (e.g., shown in FIG. 5). The kiosk 200 ispreferably located at the edge (e.g., onsite at a user facility), butcan alternatively be located in another venue.

The kiosk housing functions to define the examination space 224 (e.g.,measurement volume), and can optionally retain the sensors in apredetermined configuration about the measurement volume. The kioskhousing can optionally define one or more item insertion regions (e.g.,between housing walls, between housing arms, along the sides or top ofthe measurement volume, etc.) along one or more sides of the housing.The housing can include: a base 204 and one or more arms, wherein themeasurement volume is defined between the base and arm(s). The one ormore arms can house the processing system 580 and/or any other suitablecomponents. The one or more arms can be the support panel portion 206and/or any other components of the system.

The base 204 is preferably static relative to the arms and/or sensors,but can alternatively be mobile (e.g., be a conveyor belt). The basepreferably includes a calibration pattern, but can alternatively have nopattern, have a solid color (e.g., black), be matte, be reflective, orbe otherwise optically configured. However, the base can be otherwiseconfigured.

The calibration pattern 520 preferably functions to enable cameracalibration for the imaging device (e.g., enables the system todetermine the location of each camera with reference to a commoncoordinate system). The calibration pattern can be used to determine oneor more calibration matrices for: a single camera, a stereocamera pair,and/or any other suitable optical sensor. The calibration matrices canbe: intrinsic calibration matrices, extrinsic calibration matrixrelating the camera to the measurement volume, extrinsic matricesrelating the cameras to each other, and/or other calibration matrices.The calibration pattern is preferably arranged on (e.g., printed on,stuck to, mounted to, etc.) the base of the housing, but canalternatively be arranged along an interior wall, an arm, and/orotherwise arranged. The calibration pattern (or portions thereof)preferably appear in each optical sensor's field of view, but canalternatively appear in all RGB sensors' fields of view, a subset of theoptical sensors' fields of view, and/or otherwise appear in the images.The calibration pattern is preferably axially asymmetric (e.g., alongone or more axes, such as the x-axis, y-axis, etc.), but canalternatively be symmetric along one or more axes. The calibrationpattern can be an array of shapes (e.g., circles, squares, triangles,diamonds, etc.), a checkerboard, an ArUco pattern, a ChArUco pattern,multiple CharuCo targets (e.g., arranged as a checkerboard, grid, etc.),a circle grid pattern, an image, a logo (e.g., of the merchant), and/orany other calibration pattern. The calibration pattern can include oneor more colors (e.g., red, green, blue, and/or various shades orcombinations) and/or be black and white. The parameters of thecalibration pattern (e.g., shape size, shape arrangement, patternalignment with the measurement volume's axes, pattern pose relative tothe measurement volume, etc.) are preferably known, but canalternatively be unknown. The calibration can be raised (e.g., less than1 mm, less than 2 mm, less than 5 mm, etc.) or smooth (e.g., planar).However, the calibration pattern can be otherwise configured.

The arms 510 are preferably static, but can alternatively be actuatable.The arms can extend from the base (e.g., perpendicular to the base, at anon-zero angle to the base, etc.), extend from another arm (e.g.,parallel the base, at an angle to the base, etc.), and/or be otherwiseconfigured. The arms 510 can be arranged along the side of themeasurement volume, along an upper portion of the measurement volume,and/or otherwise arranged. Alternatively, the kiosk can not have arms.The housing can optionally include a top, wherein the top can bound thevertical extent of the measurement volume and optionally control theoptical characteristics of the measurement volume (e.g., by blockingambient light, by supporting lighting systems, etc.). However, thehousing can be otherwise configured.

The sensors 580 function to sample measurements of the items within themeasurement volume. The sensors are preferably mounted to the arms ofthe kiosk housing, but can additionally or alternatively be mounted tothe housing side(s), top, bottom, threshold (e.g., of the item insertionregion), corners, front, back, and/or any other suitable portion of thehousing. The sensors are preferably arranged along one or more sides ofthe measurement volume, such that the sensors monitor one or more viewsof the measurement volume (e.g., left, right, front, back, top, bottom,corners, etc.). The sensors can be arranged such that they collectivelymonitor a predetermined percentage of the measurement volume's points ofview (e.g., greater than 20%, greater than 50%, greater than 70%,greater than 80%, etc.), which can provide more viewing angles for anunknown item, but can alternatively monitor a smaller proportion. Thesensors can be arranged such that each imaging sensor's field of viewencompasses the calibration pattern on the base of the housing, aportion of the calibration pattern (e.g., greater than 60%, greater than70%, greater than 80%, etc.), none of the calibration pattern, and/orany other feature of the housing or portion thereof. The sensors can bearranged with an active surface directed: downward, upward, to the side(e.g., left, right, forward, back), toward a measurement volume corneror edge (e.g., left, right, back, front, top, bottom, combinationthereof, etc.), and/or otherwise arranged. In a specific example, thesensors are arranged along at least the left, right, back, and top(e.g., head edge, head central region, etc.) of the measurement volume.However, the sensors can be otherwise arranged.

The kiosk preferably includes multiple sensors, but can alternativelyinclude a single sensor. The sensor(s) can include: imaging devices(e.g., 404, 408, etc.), illumination devices (e.g., 412, 416, etc.),depth sensors, weight sensors (e.g., arranged in the base), acousticsensors, touch sensors, proximity sensors, and/or any other suitablesensor. An imaging device functions to output one or more images of theexamination space (e.g., image of the items within the measurementvolume), but can additionally or alternatively output 3D information(e.g., depth output, point cloud, etc.) and/or other information. Theimaging device can be a stereocamera system (e.g., including a left andright stereocamera pair), a depth sensor (e.g., projected light sensor,structured light sensor, time of flight sensor, laser, etc.), amonocular camera (e.g., CCD, CMOS), and/or any other suitable imagingdevice.

In a specific example, the automated object recognition kiosk 200includes stereocamera systems mounted to at least the left, right,front, and back of the measurement volume, and optionally includes atop-mounted depth sensor and/or top-mounted imaging system (e.g.,stereocamera system).

The automated object recognition kiosk 200 may be implemented indifferent business scenarios, such as for retail checkouts. For this,the automated object recognition kiosk 200 may be trained to obtain amodel using various computer vision and machine learning methods knownin the art, related art, or developed later. The obtained model may bestored in the storage device 106 and applied by the object recognitiondevice 108 for recognizing products such as one or more fresh foodsincluding, but not limited to, fresh fruits and vegetables, dairyproducts, freshly prepared eatables such as curries, breads, pastas,salads, and burgers; or any combination thereof.

In order to train the kiosk 200, the controller 102 may (1) configure apredetermined calibration pattern based on the ambient lightingconditions, (2) initialize predefined or dynamically defined attributesof the cameras and the LED lights based on the ambient lightingconditions, (3) calibrate relative positions of the cameras with respectto each other and/or at least one of the base portion 204 and theproduct; and (4) adaptively tune the LED lights to generate relativelywhite light for illuminating the base portion 204 to a predeterminedlevel of brightness, upon the kiosk 200 being switched ON. Thepredetermined brightness level of the illuminated base portion 204 maybe relatively greater than the brightness of the ambient light enteringinto the examination space 224 between the head portion 202 and the baseportion 204 of the kiosk 200. Subsequently, the automated objectrecognition kiosk 200 may be fed with details of inventory productsincluding packaged as well as fresh products in a retail store eitherdirectly through the interactive display screen 220, or via a connectionto a point of sale (POS) terminal (not shown) over the network. Thedetails may include product name, product type, price, manufacturingdate, expiry date, batch identification number, quantity, packagedimensions, etc. and may be stored in an inventory or object database inthe storage device 106.

One or more products for which the kiosk 200 need to be trained, suchproducts may be introduced within the examination space 224 by a user.In one example, one or more products such as fresh items, which may notbe covered with an opaque covering such as a package cover, a humanhand, etc., may be introduced within the examination space 224. Theproducts may be exposed to the light generated by the illuminationdevice 110 such as the LEDs 412, 416 and the imaging devices such as thecameras 404, 408. Each product may be placed in multiple positions andorientations at a predefined location such as on a predetermined regionof the base portion 204. The placed product may be directly imaged byone or more imaging devices such as the cameras 404, 408 to capturemultiple images of the products. The captured images may be stored inthe storage device 106 of the kiosk 200.

The controller 102 may be configured to feed the captured images as atraining dataset to the object recognition device 108, which may beconfigured to extract multiple features (e.g., brightness, contrast,hue, size, shape, texture, etc.) from the captured images of theproducts. The object recognition device 108 may use extracted featuresas inputs to a predetermined computer vision and machine learning methodthat may formulate an optimized model based on the extracted features.The optimized model may be saved in the storage device 106 by the objectrecognition device 108. Similarly, the automated object recognitionkiosk 200 may be trained for various package covers used to pack orcarry or hold the products, for example, the fresh items.

In order to recognize the product, the object recognition kiosk 200 maybe configured with relatively the same values for at least one of theinitialization parameters being implemented for training the kiosk 200.Examples of these initialization parameters include, but not limited to,calibration pattern, attributes of the cameras 304, 308, 404, 408 andthe LED lights 412, 416, relative positions of the cameras 304, 308,404, 408, brightness level of the LED lights 412, 416. However, in someembodiments, the values of the initialization parameters may vary fromtheir training values based on the ambient light conditions and relativepositions of the cameras 304, 308, 404, 408, the base portion 204, andthe products to be recognized.

A user may introduce one or more products within the examination space224 of the automated object recognition kiosk 200. Multiple cameras ofthe kiosk 200 may simultaneously capture multiple images of the productfrom different positions and orientations. The captured images may befed to the object recognition device 108 by the controller 102. Theobject recognition device 108 may extract multiple features from thereceived images and apply the optimized model stored in the storagedevice 106 to these extracted features for recognizing the product basedon the inventory product details stored in the storage device 106. Uponrecognizing the product, the controller 102 may provide a visual, audioor textual indication to a user. For example, the controller 102 mayprovide a pop-up message on the interactive display screen 220 with abeep to indicate a user that the product has been recognized.Additionally, the controller 102 may provide related details of therecognized product including, but not limited to, name, type, quantity,price, etc. on the display screen for the user. Some embodiments inwhich the product was placed on the kiosk base portion 204 equipped witha weight sensor, the controller 102 may display the weight of theproduct on the interactive display screen 220. In some embodiments, thecontroller 102 may provide the indication regarding the product on oneor more computing devices such as a mobile phone of the user over thenetwork. The user may use the received indication to pay for the productat a payment device such as a credit card reader, which may beintegrated with the kiosk 200, or at a POS terminal in communicationwith the kiosk 200, for completing the product purchase and the relatedtransaction. In some embodiments, the payment device or the POS terminalmay not be in communication with the kiosk 200.

In order to return a purchased product, the user may re-introduce theproduct within the examination space 224. The object recognition device108 may recognize the product using the optimized model as discussedabove and provide an indication to the user. Based on the indication, apredetermined amount may be returned to the user as per one or morepredefined criteria either directly by asking the user to swipe hiscredit or debit card against a card reader or by a cashier at the POSterminal. Examples of the predefined criteria may include, but notlimited to, the product being rescanned by the cameras may be returnedonly within two hours from the time of purchase; the package cover ofthe purchase product should not be tampered with for the product beingreturned; products may not be eligible for return after purchase, etc.

Exemplary embodiments are intended to cover all software or computerprograms capable of performing the various heretofore-discloseddeterminations, calculations, etc., for the disclosed purposes. Forexample, exemplary embodiments are intended to cover all software orcomputer programs capable of enabling processing systems to implementthe disclosed processes. In other words, exemplary embodiments areintended to cover all systems and processes that configure a computingdevice to implement the disclosed processes. Exemplary embodiments arealso intended to cover any and all currently known, related art or laterdeveloped non-transitory recording or storage mediums (such as a CD-ROM,DVD-ROM, hard drive, RAM, ROM, floppy disc, magnetic tape cassette,etc.) that record or store such software or computer programs. Exemplaryembodiments are further intended to cover such software, computerprograms, systems and/or processes provided through any other currentlyknown, related art, or later developed medium (such as transitorymediums, carrier waves, etc.), usable for implementing the exemplaryoperations disclosed above.

In accordance with the exemplary embodiments, the disclosed computerprograms may be executed in many exemplary ways, such as an applicationthat is resident in the storage device 106 of a device or as a hostedapplication that is being executed on a server or mobile computingdevice, and communicating with the device application or browser via anumber of standard protocols, such as TCP/IP, HTTP, XML, SOAP, REST,JSON and other sufficient protocols. The disclosed computer programs maybe written in exemplary programming languages that execute from memoryon the computing device or from a hosted server, such as BASIC, COBOL,C, C++, Java, Pascal, or scripting languages such as JavaScript, Python,Ruby, PHP, Perl or other sufficient programming languages.

The above description does not provide specific details of manufactureor design of the various components. Those of skill in the art arefamiliar with such details, and unless departures from those techniquesare set out, techniques, known, related art or later developed designsand materials should be employed. Those in the art are capable ofchoosing suitable manufacturing and design details.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.It will be appreciated that several of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be combined intoother systems or applications. Various presently unforeseen orunanticipated alternatives, modifications, variations, or improvementstherein may subsequently be made by those skilled in the art withoutdeparting from the scope of the present disclosure as encompassed by thefollowing claims.

We claim:
 1. A system for object identification, comprising: ameasurement volume cooperatively defined by a support component, abottom component, and a head component, wherein the support componentsupports the head component above the bottom component, wherein themeasurement volume comprises a first side defined by the supportcomponent and three substantially open sides; the head component thatencloses a top of the measurement volume, wherein the head componentcomprises: a front optical sensor that is positioned proximal a frontedge of the head component, wherein a field of view of the front opticalsensor encompasses a back edge of the measurement volume; a back opticalsensor that is positioned proximal a back edge of the head component,wherein a field of view of the back optical sensor encompasses a frontedge of the measurement volume; a left optical sensor that is positionedproximal a left side of the measurement volume, wherein a field of viewof the left optical sensor encompasses a right edge of the measurementvolume; and a right optical sensor that is positioned proximal a rightside of the measurement volume, wherein a field of view of the rightoptical sensor is encompasses a left edge of the measurement volume; anda processing system configured to automatically recognize at least onefood object using an image from at least one of the optical sensors. 2.The system of claim 1, wherein the front, back, left, and right opticalsensors comprise camera pairs.
 3. The system of claim 1, furthercomprising a first and second top optical sensor mounted to a centralregion of the head component, wherein a first and second field of viewof the first and second top optical sensors are directed downward,respectively.
 4. The system of claim 3, wherein the first top opticalsensor comprises a color camera.
 5. The system of claim 4, wherein thesecond top optical sensor comprises a 3D scanner.
 6. The system of claim1, wherein the bottom component is static relative to the head componentand comprises a calibration pattern.
 7. The system of claim 6, whereinthe calibration pattern is embedded into the bottom component.
 8. Thesystem of claim 6, wherein the calibration pattern is visible in theimage.
 9. The system of claim 6, wherein the processing system isconfigured to periodically recalibrate the optical sensors based on anappearance of the calibration pattern in images captured by the opticalsensors.
 10. The system of claim 1, wherein images from at least oneoptical sensor are used by the processing system to track a movement inthe measurement volume.
 11. The system of claim 10, wherein the supportcomponent comprises an optical sensor, wherein the optical sensor of thesupport component is configured to capture the images for tracking themovement in the measurement volume.
 12. The system of claim 1, furthercomprising a light component mounted to the head component, positionedbetween the front, back, left and right optical sensors, and wherein thelight component is configured to illuminate the measurement volume withdiffused light.
 13. The system of claim 1, wherein a lateral extent ofthe head component is coextensive with a lateral extent of the bottomcomponent.
 14. The system of claim 1, wherein the system furthercomprises a weight sensor.
 15. The system of claim 14, wherein theweight sensor is mounted to the bottom component.
 16. The system ofclaim 1, wherein the processing system is communicatively connected to aPOS terminal, wherein the processing system is automatically configuredto generate a transaction based on the recognized food object.
 17. Acheckout kiosk, comprising: a support component; a head componentcomprising a 3D scanner; a bottom component configured to receive anobject, wherein the support component statically retains the bottomcomponent below the head component; a front optical system that ispositioned proximal a front edge of the head component, wherein a fieldof view of the front optical system encompasses a back edge of thebottom component; a back optical system that is positioned proximal aback edge of the head component, wherein a field of view of the backoptical system encompasses a front edge of the bottom component; a leftoptical system, wherein a field of view of the left optical systemencompasses a right edge of the bottom component; and a right opticalsystem, wherein a field of view of the right optical system encompassesa left edge of the bottom component; and a processing system configuredto automatically recognize at least one food object using an image fromat least one of the optical systems.
 18. The system of claim 17, whereinthe bottom component comprises a calibration pattern, wherein theprocessing system is configured to periodically recalibrate the opticalsystems based on an appearance of the calibration pattern in imagescaptured by the optical systems.
 19. The system of claim 17, wherein thefront, back, left, and right optical systems comprise camera pairs. 20.The system of claim 17, wherein the processing system is communicativelyconnected to a POS terminal, wherein the processing system isautomatically configured to generate a transaction based on therecognized food object.